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Summary 


This guide describes a five-step process for selecting a sample and administering a survey. 
Educators can implement this process in their schools, districts, or states to understand a 
larger population by surveying a subgroup of its members. This approach is supported by 
the research literature on survey methodology as an accepted way to reduce costs and per- 
sonnel requirements and increase the speed of analysis and reporting while maintaining 
accuracy of the results (Cochran, 1977; Ross, 2005). Examples from a Regional Educational 
Laboratory survey project illustrate each step. 

The five-step process for selecting a sample and administering a survey is: 

• Step 1: Define the population. 

• Step 2: Specify the sampling procedure. 

• Step 3: Determine the sample size. 

• Step 4: Select the sample. 

• Step 5: Administer the survey. 

This guide is the second in a three-part series of survey method guides for educators. The 
first guide in the series covers survey development, and the third guide in the series covers 
data analysis and reporting. 
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Why this guide? 


Increasingly, state and local educators are using data to inform policy decisions (Hamilton 
et al, 2009; Knapp, Swinnerton, Copland, & Monpas-Huber, 2006; U.S. Department of 
Education, 2010). Educators need access to a wide variety of data, some of which may not 
be in their data systems. Survey data can be particularly useful for educators by offering a 
relatively inexpensive and flexible way to describe the characteristics of a population. To 
obtain such information, educators may need to conduct their own survey research. 

Survey research process 

The survey research process includes survey development, sample selection and survey 
administration, and data analysis and reporting (figure 1). The activities undertaken in 
each stage may vary depending on the type of information being collected (Fink, 2013). 
For example, if educators want to conduct a survey on a topic that has been widely sur- 
veyed in similar settings, they may be able to use an existing survey instrument and skip 
the survey development stage altogether. Or they may decide to use an existing survey but 
add items of particular interest to their context. If no surveys exist for the topic, educators 
will need to develop a new survey in stage 1 (see figure 1). 

The three guides in this series correspond to the three stages in figure 1 and provide an 
overview of survey methodologies for a nontechnical audience of educators. However, the 
resources needed to complete some of the activities may make the series most relevant to 
larger school districts, state departments of education, and other large agencies. The first 
guide in the series covers survey development (Irwin & Stafford, 2016), this second guide 
in the series covers sample selection and survey administration, and the third guide in the 
series covers data analysis and reporting (Pazzaglia, Stafford, & Rodriguez, 2016). 

These guides are intended to lead to the use of survey findings in decisions about policy 
and practice in schools and local or state education agencies; however, offering guidance 
on how decisionmakers can use the results is not the focus. 

Because this series is an overview, readers seeking a comprehensive guide to survey 
methods or to complex analyses of survey items are encouraged to explore other resources, 
such as textbooks, journal articles, and websites dedicated to these topics or to engage con- 
sultants who have this type of expertise. For useful references on selecting a sample and 
administering a survey, see appendix A. 

In providing guidance on sample selection and survey administration, this guide draws 
on examples from a survey project conducted by the authors along with educators in 
the Virtual Education Research Alliance of the Regional Educational Laboratory (REL) 


Figure 1. Three stages in the survey research process 
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Midwest about online learning in Iowa and Wisconsin (Clements, Pazzaglia, & Stafford, 
2015; Clements, Stafford, Pazzaglia, & Jacobs, 2015). See box 1 for an overview of the 
study. 

Deciding whom to survey 

For some research initiatives educators may be able to administer their survey to every 
member of the group of interest, called the target population (see box 2 for definitions of 
key terms). When this approach is not practical (for example, when surveying the entire 
target population would be too expensive or time consuming), educators may use statistical 
sampling procedures to generalize about the target population from a small sample of its 
members (Ross, 2005). Sampling procedures can reduce the costs associated with gath- 
ering data, reduce personnel requirements for administering the survey, and increase the 
speed of analysis and reporting (Cochran, 1977; Ross, 2005). 

The sampling procedure chosen depends on the objective and scope of the survey project, 
including the budget, methods of data collection, subject matter, type of respondent 
wanted, and desired accuracy of the results. Although these considerations are covered 
in the research literature and in textbooks, most texts focus on complex sampling theory 
rather than providing accessible explanations. 

This guide offers a user-friendly explanation of the basic procedures for selecting a sample 
and administering a survey using a five-step process: defining the population, specifying 
the sampling procedure, determining the sample size, selecting the sample, and adminis- 
tering the survey (figure 2). See box 1 for details of each step, along with examples from 
the REL Midwest Virtual Education Research Alliance survey project. 


Box 1. Overview of a survey project from Regional Educational Laboratory 
Midwest’s Virtual Education Research Alliance 

To identify the types of programs and policies that can support the effective use of online 
learning, state administrators and policymakers need accurate information about how and why 
schools are turning to online learning. Recognizing the potential value of this type of informa- 
tion, the Iowa Department of Education and the Wisconsin Department of Public Instruction 
collaborated with Regional Educational Laboratory Midwest’s Virtual Education Research Alli- 
ance to administer a survey to a sample of brick-and-mortar public high schools in each state. 
The survey was designed to gather information about online course use during the 2012/13 
school year, including how schools were using online courses for their students, the reasons 
they were using online courses, the challenges they faced, and the types of support they 
offered students enrolled in online courses. 
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Box 2. Definitions of key terms 


Accessible population. The portion of the target population to which the survey team has 
access to administer a questionnaire. 

Anonymous survey responses. Survey responses that cannot be linked to the survey 
respondent. 

Bias. Bias occurs when a statistic systematically misestimates a value or trait of the target 
population. Bias can stem from the way the sample is selected. All nonprobability samples 
result in biased estimates of target population values (see below). Probability samples use 
techniques to best represent the target population but can also result in biased estimates of 
target population values if response rates are low or if those who respond to the survey differ 
in meaningful ways from those who do not (see below). 

Confidence level. The degree of certainty that the statistic that would be obtained if survey 
responses were gathered from the entire population would fall within a selected margin of 
error around the sample statistic (see below). Usually expressed as a percentage, typically as 
90 percent, 95 percent, or 99 percent. 

Confidential survey responses. Survey responses that can be linked back to the survey respon- 
dent but that will not be shared outside the survey team or other designated individuals. 

Generalize. The extension of results from a survey or other study conducted on a sample to 
the larger target population. 

Margin of error. The precision or accuracy of the statistics obtained from a survey sample; 
also known as the confidence interval. A 5 percent margin of error, for example, adds 5 per- 
centage points of uncertainty on either side of a statistic obtained from the survey (statistic 
plus or minus 5 percentage points). For example, if 75 percent of sampled survey respondents 
say “yes” to a question and the margin of error is plus or minus 5 percentage points, the true 
percentage in the population that would say “yes” is estimated to fall between 70 percent and 
80 percent. 

Nonprobability sampling. Sampling procedures in which the probability of each unit in the pop- 
ulation being included in the sample is not known. As a result, it is not possible to estimate 
the accuracy of the sample statistics as approximations of the population (that is, the extent 
to which the sample statistics match the statistics that would have resulted if the entire popu- 
lation had been surveyed). 

Population size. The number of people or other units in the target population about which the 
survey team would like to generalize or make claims based on the survey results. 

Probability sampling. Sampling procedures in which each unit in the population has a known 
chance (but not necessarily an equal chance) of being included in the sample. This type of 
sampling allows researchers to estimate the accuracy of the sample statistics as approxima- 
tions of the population (that is, the extent to which the sample statistics match the statistics 
that would have resulted if the entire population had been surveyed). 

Sample. A subgroup of the target population from which data are collected and used to gener- 
alize to the larger population. 


(continued) 
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Box 2. Definitions of key terms (continued) 

Sample statistics. Values used to summarize a set of responses obtained from sampled indi- 
viduals who respond to the survey. When data are obtained using probability sampling proce- 
dures, sample statistics are designed to be generalized to the target population. 

Sampling frame. A list of the students, teachers, or schools (units) in the accessible population. 

Sampling unit. Students, teachers, schools, or some other unit or element considered for 
selection to receive a survey questionnaire. 

Stratification variable. A characteristic used to divide the population into homogenous sub- 
groups (such as grade level or school locale) when using a stratified sampling procedure. 

Survey administration. The process of distributing the survey to the people (sampled individ- 
uals) selected to receive it, encouraging them to fill it out, and collecting it. A survey can be 
administered online, with paper and pencil, over the phone, or in some other mode. 

Survey nonresponse rate. The percentage of potential respondents selected for the survey who 
do not respond. Nonresponse can occur if individuals refuse or are not able to respond to the 
survey, or if the survey team is not able to contact them to request their response. Survey non- 
response is distinguished from item nonresponse, which occurs when a respondent returns 
the survey but skips individual questions. 

Survey research A research method that uses questions to collect data. 

Target population. The entire group the survey team wants to obtain information about by gen- 
eralizing from the survey results. 


Figure 2. Five steps in selecting a sample and administering a survey 

Define the papulation 
Specify the sampling procedure 
Determine the sample size 
Select the sample 
Administer the survey 

Source: Authors' construction. 
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Step 1: Define the population 


The first step is to define the population. The target population is the entire group that the 
survey team wants to make claims about based on the survey results. The accessible pop- 
ulation is the portion of the target population to which the survey team has access. How 
the REL Midwest Virtual Education Research Alliance survey project defined the target 
and accessible populations is described in box 3. How to define the target population, the 
type of respondent (unit) to be sampled, and the accessible population are described below. 

• Consider the characteristics of the target population. 

° In defining the target population, be as specific as possible about the desired 
respondents (for example, administrators, teachers, or students), their organi- 
zation or geographic region (for example, a particular school, district, or state), 
and other potentially important characteristics (for example, student grade 
level). 

o The target population for survey research may be broad (for example, all high 
school students in a particular state) or more focused (for example, kindergar- 
ten teachers in one district). 

• Make sure that the target population matches the research questions being 
addressed. If the survey is designed for a particular population, such as grade 9-12 
teachers, it should not be administered to a different population. 

• Define the sampling unit based on the target population. This will help in select- 
ing the appropriate sampling procedure in step 2. For example, if the target popu- 
lation is all grade 9 students in a district, each student is a unit. 

• Define the accessible population. 

° Consider factors that might limit access to some members of the target popu- 
lation. For example, some parents of students in the target population may not 
provide consent for their children to participate in a student survey. 

° Avoid selecting a target population in which units in the accessible popula- 
tion are likely to differ in important ways from units in the target popula- 
tion because this will introduce bias into the survey results and jeopardize 
the survey team’s ability to draw conclusions about the target population. 
For example, a survey team that wants to draw conclusions about all grade 4 
teachers in a state but that has access to grade 4 teachers in only one district 
will not be able to generalize survey findings to the target population. In that 
case the survey team should either restrict the target population to the acces- 
sible units (grade 4 teachers in the district) or consider ways to obtain access 
to all grade 4 teachers in the state or a more representative sample of grade 4 
teachers across districts in the state. 
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Box 3. Step 1 in practice: Defining the target and accessible populations for the 
Virtual Education Research Alliance online course use survey 

To make informed decisions about the use of online learning, representatives from the Iowa 
Department of Education and the Wisconsin Department of Public Instruction needed infor- 
mation about online course use in their states. Both states defined the target population 
as public brick-and-mortar schools with students in grades 9, 10, 11, and 12, regardless of 
school structure (for example, schools serving grades 7-12 or grades 9-12). Regular, vocation- 
al, alternative, charter, and special education schools were included in the target population. 
Full-time virtual schools were excluded from the target population because the topics covered 
by the survey questions were designed for brick-and-mortar schools that use online courses to 
supplement face-to-face courses (Clements, Stafford, et al., 2015). Because the state depart- 
ments of education had access to all schools in their state, the target and accessible popula- 
tions were the same. 


Step 2: Specify the sampling procedure 


Once the target population is clearly defined, create a sampling frame and select an appro- 
priate sampling design and procedure. How the REL Midwest Virtual Education Research 
Alliance survey project selected the sampling procedure is described in box 4. 

• A sampling frame is a list of every sampling unit (person, school, or other) in the 
accessible population. The sampling frame should include any characteristics that 
may be important in selecting a sampling procedure (for example, student grade or 
school locale), as well as contact information for each person so the survey can be 
delivered and retrieved. 

• Select an appropriate sampling design and procedure. 

° Decide whether to use a probability or nonprobability sampling design. A 
probability design allows the results to be generalized to the target population, 
whereas a nonprobability design does not. 

° Consider the sampling procedures under each design that are appropriate for 
the context and purpose of the survey. Common types of probability sam- 
pling procedures are simple random sampling, systematic sampling, stratified 
sampling, and cluster sampling (box 5). Common types of nonprobability 
sampling procedures are convenience sampling, judgment sampling, and quota 
sampling (box 6). 

o If survey results are to be generalized from a sample to a target population, a 
probability sampling design must be used. Consider the advantages and disad- 
vantages of each probability sampling procedure described in box 5. 
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Box 4. Step 2 in practice: Selecting a sampling procedure for the Virtual Education Research Alliance 
online course use survey 

Because the goal of the Virtual Education Research Alliance project was to obtain accurate statewide estimates of 
online course use, the study team opted to use a probability sampling design that would allow the results to be gen- 
eralized to the target population. Results of previous studies indicated that the use of online learning differs across 
urban, town, and rural districts (Picciano & Seaman, 2009; Queen & Lewis, 2011), so the study team chose a strat- 
ified sampling procedure. The study team stratified the target population by school locale (city or suburban, town, or 
rural) and selected a random sample of schools from each locale subgroup in proportion to the size of the subgroup 
(that is, the percentage of schools sampled from each locale type matched the statewide percentage of schools 
in that type of locale). This sampling procedure ensured that the Iowa and Wisconsin samples were representative 
of the target populations with respect to school locale. To the extent that school locale is related to the measured 
online learning outcomes, this procedure improves the accuracy of the survey statistics relative to simple random 
sampling (Clements, Stafford, et al., 2015). 


Box 5. Probability sampling: Commonly used procedures and considerations 

Probability sampling requires that each unit in the population have a known chance (but not necessarily an equal 
chance) of being included in the sample. This type of sampling allows the survey team to estimate the accuracy of 
the sample statistics as approximations of the population (that is, the extent to which the sample statistics match 
the statistics that would have resulted if the entire population had been surveyed). 

1 Procedure 

Description 

Considerations t 

Simple random 
sample 

Units are randomly selected from the population 
such that each unit has an equal probability of being 

selected. 

This procedure requires access to a complete list of 
the target population. When conducted appropriately, 
random sampling results in unbiased estimates of the 
population, which means that the survey team can be 
confident that the survey results obtained from the 
sample will generalize to the larger population. 

Systematic 

sample 

Consecutive numbers are assigned to each unit in the 
population, a starting number is randomly selected, and 
then units are selected at regular intervals from the list 
to obtain the desired sample size (for example, every 10 
units). 

This procedure requires access to a complete list of 
the target population that does not include an order or 
pattern that would bias the sample (for example, if male 

and female students were alternated on the list and the 

sample interval was an even-numbered unit, only one 
gender would be selected). 

Stratified 

sample 

Homogeneous subgroups in the population (for example, 
urban schools and rural schools) are selected, and units 
are randomly selected from each subgroup, usually in 
proportion to the size of the subgroup. 

This procedure requires access to a complete list of 
the target population. If the stratification variable is 
related to the survey outcomes of interest, stratification 
provides increased accuracy of estimates that can 
be generalized to the population without substantial 

increases in costs. 

Cluster sample 

Units are randomly selected from clusters (for example, 
classrooms) in the target population. Either all units in 
each cluster (for example, all students in the selected 
classrooms) are included or a random sample of units 
from each cluster (for example, a sample of students in 
the selected classrooms) is included. 

This procedure requires access to a complete list of 
the target population of clusters and units within the 
selected clusters. It can reduce the cost of surveying 
(for example, rather than surveying one student from 
each of 100 different classrooms, a survey team might 
sample 20 students from each of 5 classrooms). 

Source: Ross, 2005. 
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Box 6. Nonprobability sampling: Commonly used procedures and considerations 

With nonprobability sampling the probability of each unit in the population being included in the sample is not 
known. As a result, it is not possible to estimate the accuracy of the sample statistics as approximations of the pop- 
ulation (that is, the extent to which the sample statistics match the statistics that would have resulted if the entire 
population had been surveyed). Despite this limitation, nonprobability sampling procedures can be appropriate in 
some situations, including small-scale testing or piloting surveys prior to full administration. 

1 Procedure 

Description 

Considerations |i 

Convenience 

sample 

Units are selected from the population based on their 
accessibility to the survey team. 

This procedure limits the time and costs involved in 
small-scale survey administrations and can result in 
increased response rates. However, in many cases, 
easily accessible units of the population are likely to 
differ from less accessible units in important ways. This 
introduces bias into the survey results, which means 
that sample estimates are less likely to generalize to 
the larger population. 

Judgment 

(purposive) 

sample 

An effort is made to select a “typical’’ sample that 
is representative of the population to the best of the 
survey team’s knowledge. 

This procedure can reduce time and costs. The quality 
of the sample depends on the survey team’s ability 
to select a sample that is typical of the population. 
Different survey teams will likely disagree on what 
constitutes a typical sample, and there is no way 
to determine how well the results generalize to the 
population. 

Quota sample 

Units are selected from various population subgroups in 
proportion to the size of the subgroup. 

This procedure controls the proportion of units sampled 
from each subgroup, which can increase the accuracy of 
estimates if conducted appropriately. However, there is 
typically little control over the procedures for sampling 
within subgroups, as either convenience or judgment 
may determine the sampling. As a result, there is no 
way to check the accuracy of estimates. 

Source: Ross, 2005. 


Step 3: Determine the sample size 


Step 3 is to determine how many students, teachers, schools, or other units need to be 
sampled using the procedure selected in step 2. 

For probability sampling procedures, sample size tables, such as the one shown in table 1, can be 
used to quickly estimate the required sample size 

To use a sample size table, first determine the target population size, the desired margin 
of error and confidence level, and the anticipated nonresponse rate, as described below. 
How the REL Midwest Virtual Education Research Alliance survey project determined 
the sample size is described in box 7. 

• Take the accessible population size from the sampling frame created in step 2. If 
the population size falls between the values provided in table 1, round up to the 
next number for a conservative estimate. 

• Select the margin of error; the margin of error estimates the precision or accuracy 
of the statistics obtained from a survey sample. A 5 percent margin of error is 
the most common; it adds 5 percentage points of uncertainty around the sample 
statistic (statistic plus or minus 5 percent). Smaller margins of error may be chosen 
for a critical policy decision that requires precise estimates. 
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Table 1. Sample size table for determining the minimum sample sizes required to achieve the specified 
confidence levels and margins of error 

Population size 

Number of units needed for a 
confidence level of 95 percent 

Number of units needed for a 
confidence level of 99 percent 

At a 5 percent 
margin 
of error 

At a 2.5 percent 
margin of error 

At a 1 percent 
margin 
of error 

At a 5 percent 
margin of error 

At a 2.5 percent 
margin of error 

At a 1 percent 
margin 
of error 

10 

10 

10 

10 

10 

10 

10 

20 

19 

20 

20 

19 

20 

20 

50 

44 

48 

50 

47 

49 

50 

100 

80 

94 

99 

87 

96 

99 

500 

217 

377 

475 

285 

421 

485 

1,000 

278 

606 

906 

399 

727 

943 

10,000 

370 

1,332 

4,899 

622 

2,098 

6,239 

100,000 

383 

1,513 

8,762 

659 

2,585 

14,227 

500,000 

384 

1,532 

9,423 

663 

2,640 

16,055 

1,000,000 

384 

1,534 

9,512 

663 

2,647 

16,317 

Source: Authors' calculations based on a formula presented in Krejcie and Morgan (1970). 


• Select the desired confidence level; the confidence level indicates the certainty that 
the sample results reflect the results for the target population. The most common- 
ly chosen confidence level is 95 percent, which implies a 95 percent certainty that 
the target population value would fall within the sample statistic, plus or minus the 
selected margin of error. A 99 percent confidence level may be selected if the infor- 
mation is for a critical policy decision for which users want to be extremely confident 
that the survey results reflect the views or characteristics of the larger population. 

• Correct the required sample size for expected nonresponses. Sample size tables like 
table 1 show the number of respondents needed to achieve a given level of con- 
fidence that the results will reflect the views of the larger population. Because it 
is rare to get responses from 100 percent of the sampled units, it is necessary to 
adjust the sample size for anticipated nonresponses. Anticipated nonresponse rates 
can be predicted using response rates from similar previous surveys or the survey 
team’s knowledge of the sample’s willingness or ability to respond (for instance, 
response rates will likely be lower for parents than for teachers). 

For example, if only half of those invited to complete the survey are likely to 
respond, the number obtained from table 1 should be divided by the anticipated 
response rate, or 0.5, to get the final sample size. Thus, if the table indicates that 
80 responses are required, dividing 80 by 0.5 would give a final sample size of 160. 

This procedure corrects for potential nonresponse by adding units to the 
sample. However, if the anticipated response rate is low enough, it is possible to 
calculate a required sample size that exceeds the target population size. In this case, 
the team should consider strategies for increasing the response rates (see step 5). 

A low response rate can limit the ability to generalize the sample results to the 
target population. If the survey response rate is less than 85 percent for a probability 
sample, the results may not be representative of the target population (U.S. Depart- 
ment of Education, 2012). In this case, it is necessary to check for potential bias (see 
the third guide in this series, Pazzaglia et al., 2016). Based on this guidance, one 
approach is to select 85 percent as a desired response rate to limit the risk of bias and 
initiate the strategies discussed in step 5 to get the highest response rate possible. 
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Box 7. Step 3 in practice: Determining the sample size for the Virtual Education 
Research Alliance online course use survey 

For both Iowa and Wisconsin the desired margin of error was 5 percent, the confidence level was 
95 percent, and the anticipated response rate was 85 percent. The margin of error and confidence 
level were chosen because they are the standard values in the literature, and the response rate 
was selected based on the desire to reach the U.S. Department of Education’s standards (2012). 
In both states, the population was stratified by school locale: city or suburban, town, or rural. 

For Iowa, the stratified sample calculations were as follows (see table below for the final 
sample sizes based on the following calculations): 

• Population size: 396 schools. 

• Required sample size using table 1: 217 total schools. 

• Required sample size correcting for nonresponse: 217/0.85 = 256 total schools. 

• Required sample size for each subgroup: 

° City or suburban: 256 * 0.13 (proportion of city or suburban schools in population) = 
33 schools. 

o Town: 256 * 0.23 (proportion of town schools in population) = 59 schools. 

° Rural: 256 * 0.64 (proportion of rural schools in population) = 164 schools. 

For Wisconsin, the stratified sample calculations were as follows (see table below for final 
sample sizes): 

• Population size: 605 schools. 

• Required sample size using table 1: 278 total schools. 

• Required sample size correcting for nonresponse: 278/0.85 = 327 total schools. 

• Required sample size for each subgroup: 

° City or suburban: 327 * 0.34 (proportion of city or suburban schools in population) = 
111 schools. 

° Town: 327 * 0.19 (proportion of town schools in population) = 62 schools. 

° Rural: 327 * 0.47 (proportion of rural schools in population) = 154 schools. 


Population and stratified samples for the Virtual Education Research Alliance online course 
use survey 


Locale 


Iowa 



Wisconsin 


Population 

Percent 

Sample size 

Population 

Percent 

Sample size 

City or suburban 

51 

13 

33 

207 

34 

111 

Town 

92 

23 

59 

112 

19 

62 

Rural 

253 

64 

164 

286 

47 

154 

Total 

396 

100 

256 

605 

100 

327 


Note: The sample sizes reported here do not correspond to the estimates reported in Clements, Stafford, et al. 
(2015) because the study team used a more complex formula specific to that project (see Cochran, 1977, for 
more details); however, table 1 provides a useful generalization that can be used for any probability sampling 
procedure, though the resulting sample sizes may be conservative (larger than necessary) in some cases. 

Source: Adapted from Clements, Stafford, et al. (2015). 
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For nonprobability sampling procedures, practical considerations rather than statistical considerations 
often guide the sample size selection because the results cannot be generalized to a target population 

The following considerations can help determine the sample size: 

• Use sample size tables as a general reference for nonprobability samples if the size 
of the target population is known. 

• Consider the number of units that are accessible or feasible to survey given the 
personnel and cost limitations. 

Step 4: Select the sample 


After selecting a sampling procedure and determining the required sample size, select the 
units to whom the survey will be administered. Steps for probability and nonprobability 
sampling designs are described below. The procedure that the REL Midwest Virtual Edu- 
cation Research Alliance survey project used to select sampling units is described in box 8. 

For probability sampling procedures several steps might be included to select the sample 

• Use either a random number generator or a random number table to select units 
from the sampling frame. Screenshots depicting the steps required to use the 
random number generator function in Microsoft Excel to obtain a random sample 
are shown in appendix B. 

• Compare the characteristics of the selected sample to the characteristics of the 
target population to ensure that the sample is similar to the population. The char- 
acteristics chosen can be based on research literature on the topics being assessed 
by the survey (for example, previous research suggests that the use of online 
learning differs across grade levels) or interests of the survey team (for example, 
state departments of education may be interested in differences in online course 
use based on school locale). If there is evidence that the sample is not similar 
to the population in some way (for example, the proportion of rural schools in 
the sample is much lower than the proportion of rural schools in the population), 
consider using a stratified sampling procedure or analysis procedures that correct 
the survey estimates by weighting the responses of particular types of respondents 
(for example, respondents from rural schools). Part 3 of this series provides more 
information about these procedures. 

• Create a new list that contains only the sampled units, any stratification variables 
or characteristics that may be used to calculate survey statistics for subgroups of 
respondents (for example, students in different grades), and contact information 
for each unit (see step 4 in appendix B for an example). 

For nonprobability sampling procedures several steps might be included to select the sample 

• Determine the most accessible units of interest. For example, consider targeting 
the schools of individuals with whom you have personal or professional connec- 
tions because those individuals may be able to encourage other individuals in their 
schools to complete the survey. 

• Get informed recommendations about typical population characteristics or strati- 
fication variables from the literature or experts in the field. 

• Create a list of sampled units, any stratification variables or characteristics that 
may be used to calculate survey statistics for subgroups of respondents, and contact 
information for each unit (see step 4 in appendix B for an example). 
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Box 8. Step 4 in practice: Selecting the sample for the Virtual Education Research 
Alliance online course use survey 

To select the stratified sample for the REL Midwest Virtual Education Research Alliance’s 
study, the study team first created Microsoft Excel files for Iowa and Wisconsin that listed 
every brick-and-mortar public high school in the target population along with their locale des- 
ignation, contact information, and other characteristics of interest to the state departments 
of education that were available in existing data sources (locale, Title 1 eligibility [yes or no], 
school type [regular, alternative/vocational, special education], and grade span [7-12, 9-12, 
other]). Separate tabs were created for city or suburban, town, and rural schools in each state 
file. Next, the project team used the random number generator function in Excel [=RAND()] 
to create a column of random numbers with the same number of entries as the number of 
schools in each tab. 

The project team sorted the school lists by the random number column and took the 
required number of schools in each locale category from the top of each list (for example, 
the first 33 schools in the Iowa city or suburban tab). A new file was created for each state 
that contained only the sampled schools across locale categories to guide administration and 
follow up (see step 5). Finally, the study team confirmed that the sample aligned with the 
population proportions in terms of locale, Title 1 eligibility, school type, and grade span. See 
appendix B for screenshots depicting the steps for using Excel to obtain random samples and 
an example of the spreadsheet created for this project. 


Step 5: Administer the survey 

Before administering the survey to the final sample, decide how long to continue to collect 
data, how frequently and in what manner to follow up with nonresponders, and how to 
encourage participation. How the REL Midwest Virtual Education Research Alliance 
administered its survey project is described in box 9. Considerations and recommendations 
gleaned from that survey project include the following: 

• Gain approvals at the school, district, and state levels, as appropriate, prior to 
administration. Complete any institutional review board or consent process- 
es required by the local or state education agency. Approval processes can take 
months in some cases, so allow sufficient time for this step. 

• Consider strategies to get the highest response rates possible. If the survey response 
rate is less than 85 percent in a probability sample design, the results may not be 
representative of the target population (U.S. Department of Education, 2012). See 
the third guide in this series (Pazzaglia et al., 2016) for information about how to 
check for representativeness. 

o Allow anonymous or confidential responses rather than identifiable respons- 
es whenever possible to increase participants’ willingness to respond to the 
survey and disclose information. 

° Consider incentives for participation (for example, gift cards or entrance into 
a drawing for a prize). 

o Consider sending advance letters to key personnel (for example, superinten- 
dents of districts being surveyed) prior to administering the survey to explain 
the purpose and potential value of the survey, 
o Enlist a trusted colleague of the potential respondents or a respected organi- 
zation to send endorsement letters, invitations, and reminders (for example, 
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the state departments of education sent invitations and reminders for the REL 
Midwest Virtual Education Research Alliance survey project). 

° Build awareness of the survey and of the importance of the resulting data 
through websites, posters, announcements, and newsletters, 
o Offer time for individuals to voluntarily complete surveys during a scheduled 
activity along with the necessary resources to do so (tablets, laptops, or hard 
copies of the survey). 

• Consider data security. 

o Many free online survey platforms allow data to be stored on the software 
company’s server, which could be problematic if personally identifiable infor- 
mation or other sensitive information is collected, 
o For a paper-and-pencil survey plan for how to secure completed surveys, such 
as storing them in a locked filing cabinet. 

• Carefully consider the time when the survey is administered. Avoid busy times 
of the year for potential respondents such as the beginning or end of the school 
year or holidays. Provide enough time (often six to eight weeks) to yield desired 
response rates, and consider extending the survey administration period if the 
desired response rate is not achieved by the deadline. 

• Communicate clearly and efficiently to potential respondents. 

o In an invitation email or letter cover the key elements, including a brief, clear 
description of what the survey is about, who is administering it, why it is import- 
ant, how much time it will take to complete, when survey responses are due, how 
the results will and will not be used, and whether responses will be kept anony- 
mous or confidential. A sample invitation letter is provided in appendix C. 

° Send brief reminders to nonresponders through emails, letters, or telephone 
calls. Be respectful by limiting the number of reminders and promptly remov- 
ing respondents from follow-up lists if requested or when they return their 
questionnaire. A sample reminder letter is provided in appendix D. 
o Thank respondents for their participation and, if possible, share a summary of 
the survey results with them. 

• Conduct an early check of data during the first week of survey administration to 
ensure that all information is being recorded correctly. 


Box 9. Step 5 in practice: Administering the Virtual Education Research Alliance 
online course use survey 

The Iowa Department of Education and the Wisconsin Department of Public Instruction admin- 
istered surveys electronically through a secure online survey system over eight weeks. The 
intended respondent at each school was the staff person most familiar with the school’s online 
learning program. The Iowa Department of Education and the Wisconsin Department of Public 
Instruction sent principals in the sampled schools an email that described the purpose of the 
study and asked them to forward the email to the staff member responsible for overseeing the 
online learning program at their school. Reminder emails were sent approximately every two 
weeks during the next eight weeks to principals in schools that had not completed the survey. 
Each nonresponding school also received up to two reminder phone calls during the final three 
weeks of the survey administration (Clements, Stafford, et al., 2015). 


13 


Using this guide 


Educators can use this guide (the second in a three-part series) to define their target popula- 
tion, specify an appropriate sampling procedure, determine the required sample size, select the 
sample, and successfully administer a survey. In combination, these steps can help educators 
develop useful generalizations about their target population by studying a smaller subgroup of 
its members. Sound sampling and administration procedures can reduce the time and costs of 
the survey team and can contribute to their ability to draw valid conclusions from the result- 
ing data. The first guide in the series covers survey development (Irwin & Stafford, 2016), and 
the third guide in the series covers data analysis and reporting (Pazzaglia et al, 2016). 

Limitations of this guide 


This guide has two limitations. First, it is intended to provide educators and policymakers 
with an overview of considerations for selecting a sample and administering surveys in 
practical education contexts; thus, it does not provide a complete review of the complex, 
technical details of sampling procedures. Educators and policymakers interested in more 
complex sampling procedures or a more detailed explanation of any sampling procedure 
may seek additional resources from a list provided in appendix A. 

Second, this guide does not provide tools, such as survey administration platforms, that 
may be needed to administer surveys. Many of these resources are available on the Inter- 
net, but it is important to consider data security when using publicly available resources. 
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Appendix A. Additional survey sampling and administration resources 


Additional resources for sampling and survey administration are provided in this appen- 
dix, including textbooks, journal articles, and professional organizations with expertise in 
these topics. 

Potentially useful documents 

Bartlett, J. E., Kotrlik, J. W., & Higgins, C. C. (2001). Organizational research: Determin- 
ing appropriate sample size in survey research. Information Technology, 19(1), 43-50. 

DeVellis, R. F. (2012). Scale development: Theory and applications (3rd ed.). Thousand Oaks, 
CA: Sage. 

Fink, A. (2013). How to conduct surveys: A step-by-step guide (5th ed.). Thousand Oaks, 
CA: Sage. 

Fowler Jr., F. J. (2008). Survey research methods (4th ed.). Thousand Oaks, CA: Sage. 

Groves, R. M., Fowler Jr., Couper, M. R, & Lepkowski, J. M. (2009). Survey methodology. 
New York, NY: Wiley. 

Hinkin, T. R. (1998). A brief tutorial on the development of measures for use in survey 
questionnaires. Organizational Research Methods, 2(1), 104-121. 

Schrijver, W. (2013). How to estimate your population and survey sample size? (Web log 
comment). Retrieved August 13, 2015, from https://www.checkmarket.com/2013/02/ 
how'tO'estimate'your'population'and'Survey'Sample'Size/. 

Wright, P. V., & Marsden, J. D. (2010). Handbook of survey research. Bingley, UK: Emerald 
Group Publishing. 

Other potentially useful resources 

AdvancED’s guide to administering surveys and generalizing survey results 
http://extranet.advanC'ed.org/assist_resources_and_tools/docs/ASSIST_SurveyGuide.pdf 

Dartmouth Office of Institutional Research’s surveys: Why, when and how 
http://www.dartmouth.edu/~library/biomed/services/lgr/docs/SurveyLGR_final.pdflmswitch 
'redir=classic 

Harvard University Program on Survey Research: Resources 
http://psr.iq.harvard.edu/book/psr'resources 
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Appendix B. Using Microsoft Excel to obtain a random sample 


Steps for obtaining a random sample in a hypothetical example are depicted below using 
screenshots from Microsoft Excel. In this example, the target population is 30 schools, and 
the required sample size is 20 schools. Random samples can also be obtained using random 
number tables or random number generators that are available online. 


Step 1: Create a Microsoft Excel spreadsheet listing every unit in the accessible population along 
with any stratification variables and contact information 
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Step 2: Use the random number generator function in Excel to create a column of random numbers 
with the same number of entries as the number of units in the file 

• Make a column heading for “Random Number.” Type =RAND() in the first cell in 
the random number column and hit Enter. 
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• The =RAND() cell will become a random number between 0 and 1. Copy this cell 
and paste the contents into all cells below it so that each unit row has a random 
number entry. 



F2 ; fx 0.478819709492506 



A 

1 B 

C 

D 

E 1 

F 

1 C 

1 

Unit 

Locale 

School Type 

Title 1 Eligibility 

Contact Info 

Random Number 

2 

School 1 

Town 

Special Education 

No 

XXX-XXX-XXXX 

0.478819709 


3 

School 2 

Town 

Regular 

Yes 

XXX-XXX-XXXX 

0.217114018 



School 3 

City 

Regular 

Yes 

XXX-XXX-XXXX 

0.844072879 



School 4 

City 

Regular 

No 

XXX-XXX-XXXX 

0.716789928 



School 5 

City 

Special Education 

No 

XXX-XXX-XXXX 

0.666809485 



School 6 

Town 

Regular 

Yes 

XXX-XXX-XXXX 

0.342088421 



School 7 

Rural 

Regular 

Yes 

XXX-XXX-XXXX 

0.711641574 



School 8 

Town 

Regular 

No 

XXX-XXX-XXXX 

0.520813586 


10 

School 9 

Rural 

Alternative 

Yes 

XXX-XXX-XXXX 

0.090965589 


11 

School 10 

City 

Regular 

Yes 

XXX-XXX-XXXX 

0.855930992 


12 

School 11 

City 

Special Education 

Yes 

XXX-XXX-XXXX 

0.985165952 


13 

School 12 

Town 

Regular 

No 

XXX-XXX-XXXX 

0.091339702 


14 

School 13 

Rural 

Regular 

No 

XXX-XXX-XXXX 

0.609186839 


15 

School 14 

Town 

Special Education 

No 

XXX-XXX-XXXX 

0.415768978 


16 

School 15 

Rural 

Regular 

No 

XXX-XXX-XXXX 

0.376723996 



Step 3: Sort the units by the random number column 

• Select all of the data by clicking and dragging across the entire range of data. 
Under the Data tab, Sort & Filter option, choose “Custom Sort. . .” 
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Check the box for “My list has headers,” and under “Sort by” select “Random 
number” column. Click OK. 
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Step 4: Copy the number of units recommended by your sample size table from the top of the list 
(the first 20 schools in this example) and paste it into a new spreadsheet that contains only the 
sampled schools along with their characteristics of interest and contact information 


A Home 

Layout 

Tables 

Charts 

SmartArt 

Formulas Data Re 

Sort & Filter 


Analysis 


External Data Sources 


▼ 1- 

Ih 

rffl 'V 

T_l' 

H ffi 

& !i i=a 

Sort Filter 

PivotTable What-lf 

Refresh 

Text Database HTML FileMaker i UtP Valid 


A2 


fx School 24 



Town Special Education No 

Rural Regular 

Rural Alternative 

Town Regular 

Rural Alternative 

Town Regular 

Town Regular 


No 

Yes 

No 

Yes 

No 

Yes 

Yes 


XXX-XXX-XXXX 

XXX-XXX-XXXX 

XXX-XXX-XXXX 

XXX-XXX-XXXX 

XXX-XXX-XXXX 

XXX-XXX-XXXX 

XXX-XXX-XXXX 

XXX-XXX-XXXX 


10 

School 6 

Town 

Regular 

Yes 

XXX-XXX-XXXX 

0.342088421 

xi 

School 29 

Rural 

Regular 

Yes 

XXX-XXX-XXXX 

0.342924955 

12 

School 15 

Rural 

Regular 

No 

XXX-XXX-XXXX 

0.376723996 

13 

School 14 

Town 

Special Education 

No 

XXX-XXX-XXXX 

0.415768978 

14 

School 19 

Rural 

Regular 

Yes 

XXX-XXX-XXXX 

0.415974132 

15 

School 25 

Town 

Regular 

Yes 

XXX-XXX-XXXX 

0.457039605 

16 

School 21 

Rural 

Alternative 

No 

XXX-XXX-XXXX 

0.478524864 

17 

School 1 

Town 

Special Education 

No 

XXX-XXX-XXXX 

0.478819709 

18 

School 8 

Town 

Regular 

No 

XXX-XXX-XXXX 

0.520813586 

19 

School 13 

Rural 

Regular 

No 

XXX-XXX-XXXX 

0.609186839 

20 

School 28 

Rural 

Alternative 

Yes 

XXX-XXX-XXXX 

0.656204284 

21 

School 5 

City 

Special Education 

No 

XXX-XXX-XXXX 

0.666809485 

22 

School 22 

City 

Alternative 

Yes 

XXX-XXX-XXXX 

0.678699815 

23 

School 17 

City 

Regular 

No 

XXX-XXX-XXXX 

0.711366591 

24 

School 7 

Rural 

Regular 

Yes 

XXX-XXX-XXXX 

0.711641574 

25 

School 4 

City 

Regular 

No 

XXX-XXX-XXXX 

0.716789928 

26 

School 18 

Town 

Regular 

No 

XXX-XXX-XXXX 

0.763255678 


Random Number 
0.034352237 
0.086188438 
0.090965589 
0.091339702 
0.157601694 
0.1703012 
0.217114018 
0.234130278 


V/V/V/ \J\/\J v/v/v/v/ 


B-3 







hi i I O O ( fa 1 





B-4 


Appendix C. Sample survey invitation 


A survey invitation from the Regional Educational Laboratory Midwest’s Virtual Educa- 
tion Research Alliance survey project is shown below as an example. This invitation was 
emailed to the principals of all sampled high schools at the start of the survey administra- 
tion period. 

Dear Principal, 

You are invited to participate in a survey regarding online course use in your school. This 
study is being conducted by [INSERT ORGANIZATION] with the support of [INSERT 
COLLABORATING ORGANIZATIONS]. The results of the study will help us better 
understand how online courses are being used in [INSERT STATE] schools. 

Below is the link to the survey, which should be completed by the staff person in your 
school who is most familiar with your online learning program. While several staff may 
need to be consulted in order to ensure accurate responses, we ask that only one represen- 
tative from your school submit the survey. Please have the following information before 
you begin the survey: 

• The number of online courses taken by students in your school in 2012/13. 

• The number of students in your school who were enrolled in online courses in 
2012/13. 

• The number of enrollments in online courses for your school in 2012/13, both 
overall and by academic subject. 

• The reasons that online courses were used. 

• How your school supported students who took online courses. 

[INSERT ORGANIZATIONS] will neither penalize nor reward you or your school based 
on your responses to the survey. The survey will take approximately 15 minutes to com- 
plete. Your school’s participation is voluntary. 

Click the link below to access the survey. 

[INSERT SURVEY LINK] 

Please complete the survey by [INSERT DATE]. 

Sincerely yours, 

[INSERT SIGNATURE AND AFFILIATION] 
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Appendix D. Sample survey reminder 


A reminder from the Regional Educational Laboratory Midwest Virtual Education 
Research Alliance survey project is shown below as an example. This reminder was emailed 
to the principals of the sampled high schools who had not completed the survey. A version 
of this email was sent every two weeks during the eight-week survey administration period. 

Hello, 

Time is running out for your school to complete the online course use survey. The results 
of the study will help us better understand how online courses are being used in [INSERT 
STATE] schools. Your participation is voluntary but vital since only a sample of schools 
have been invited to participate. 

Below is the link to the survey, which should be completed by the staff person in your 
school who is most familiar with your online learning program. The survey will take 
approximately 15 minutes to complete. 

[INSERT SURVEY LINK] 

Please complete the survey by [INSERT DATE]. 

We will be in touch via phone next week. 

Sincerely yours, 

[INSERT SIGNATURE AND AFFILIATION] 
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The Regional Educational Laboratory Program produces 7 types of reports 



Making Connections 

Studies of correlational relationships 


Making an Impact 

Studies of cause and effect 


What’s Happening 

Descriptions of policies, programs, implementation status, or data trends 


What’s Known 

Summaries of previous research 


Stated Briefly 

Summaries of research findings for specific audiences 


Applied Research Methods 

Research methods for educational settings 


Tools 

Help for planning, gathering, analyzing, or reporting data or research 





